GitHub Action for non-UTF-8 locales by aitap · Pull Request #7821 · Rdatatable/data.table

aitap · 2026-07-18T17:17:36Z

Following #7681 (comment): test data.table in the Latin-1 locale (where CE_NATIVE strings should be byte-to-byte equal to CE_LATIN1), GB18030 (which is fully Unicode-compatible, but the mapping from code points to byte sequences is very non-uniform), KOI8-R (which can represent some math symbols but not extended Latin or CJK).

codecov · 2026-07-18T17:37:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.01%. Comparing base (94649d5) to head (060f490).
⚠️ Report is 11 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7821   +/-   ##
=======================================
  Coverage   99.01%   99.01%           
=======================================
  Files          88       88           
  Lines       17234    17234           
=======================================
  Hits        17065    17065           
  Misses        169      169

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

MichaelChirico

I think it could be merged as-is, PTAL at the feedback and see what you agree is worth addressing. Thanks!

MichaelChirico · 2026-07-27T17:09:41Z

BTW, in #7832 (R-CMD-check-occasional GHA), we also run lv_LV and zh_CN locales, and are finding a number of CI errors -- I'm surprised they aren't throwing here, could you explain the discrepancy?

aitap · 2026-07-28T15:16:10Z

Here I'm running tests with only LC_CTYPE set in order to catch encoding bugs; LC_COLLATE stays unchanged. Meanwhile, Latvian conveniently uses the Latin script and defines a complex collation sequence on their alphabet. The same will happen with Azerbaijani, Turkish, Czech, Danish, Norwegian, Estonian, Hungarian, and Lithuanian, but not Spanish, Swedish, Finnish, Polish, Icelandic, French, or German.

(Try installing locales-all and feeding multi-character ASCII strings to the following function:

function(locale, strings) {
 old_collate = Sys.getlocale('LC_COLLATE')
 Sys.setlocale('LC_COLLATE', locale)
 on.exit(Sys.setlocale('LC_COLLATE', old_collate), add = TRUE)
 old_ctype = Sys.getlocale('LC_CTYPE')
 Sys.setlocale('LC_CTYPE', locale)
 on.exit(Sys.setlocale('LC_CTYPE', old_ctype), add = TRUE)
 data.table::data.table(strings, locale = base::order(strings), forderv = data.table:::forderv(strings))
}

)

MichaelChirico · 2026-07-28T16:19:51Z

Yea, lv_LV was specifically chosen for the collation rules (which indeed caught some issues in the tests).

I'm wondering if we should combine the efforts here with the R-CMD-check-occasional GHA, just add more encodings over there, add a collation check encoding on the every-push GHA, or just leave things as-is (i.e. with these two PRs unchanged), WDYT?

aitap · 2026-07-28T20:00:40Z

It's probably fine to move the encoding tests into the R-CMD-check-occasional action, together with the LC_COLLATE tests.

MichaelChirico · 2026-08-05T22:31:04Z

OK, moved the changes there. The behemoth grows :)

…sses too

ben-schwen · 2026-08-06T08:55:59Z

          - os: macOS-latest
            locale: 'zh_CN.utf8'
+          - os: macOS-latest
+            locale: 'fr_CA.ISO-8859-1'


wont we have to use 'fr_CA' in the exclusion as well? same for the exclusion of windows

ben-schwen

LGTM besides the potential clash of 'fr_CA.ISO-8859-1' and 'fr_CA'

aitap · 2026-08-06T09:29:13Z

                 # Multibyte characters: Mandarin
                 'zh_CN.utf8',
+                 # Encoding: non-UTF-8 locales for French, Mandarin, and Russian
+                 'fr_CA', 'zh_CN.GB18030', 'ru_RU.KOI8-R', # fr_CA is implicitly 'ISO-8859-1'


It's not completely reliable to rely on the implicit default encoding. It's possible to have a UTF-8 fr_CA locale with the right settings in /etc/locale.gen:

echo 'fr_CA UTF-8' | sudo tee -a /etc/locale.gen sudo locale-gen --keep-existing LANG=fr_CA Rscript -e 'l10n_info()'

$MBCS [1] TRUE $`UTF-8` [1] TRUE $`Latin-1` [1] FALSE $codeset [1] "UTF-8"

aitap · 2026-08-06T09:35:26Z

+            sudo tee "$target" << EOF
+          #!/bin/bash
+          set -o pipefail
+          exec "${target}.orig" "\$@" 2>&1 | iconv -c -t UTF-8


This will cause child R processes run by R CMD check to encode their output to UTF-8 as well. Then the overall output from the parent process will be encoded once again, with some outputs requiring echo "$unicode_text_from_browser" | iconv -t $source_encoding | iconv -t $source_encoding to read them as UTF-8 (implying some encoding from $source_encoding to UTF-8 has happened thrice, some even more).

Yea, I got as far as that, but not fixing it.

In ru_RU.KOI8-R, it looks like the root issue is {bit} not installing, possibly because of UTF-8 in R/ comments like:

https://github.com/r-lib/bit/blob/d128f0735f3c15ac67070a1398fea3411de5eab8/R/zzz.R#L2

This per LLM:

The Recursive Mojibake (Why the text is so garbled)

Child Process: Encounters an error and correctly outputs Russian text in KOI8-R (e.g. Предупреждение). Its iconv wrapper converts this to UTF-8.

Parent Process (R CMD INSTALL): Captures that UTF-8 text and echoes it to the console. The parent's iconv wrapper intercepts these UTF-8 bytes, incorrectly assumes they are KOI8-R, and converts them to UTF-8 again.

Grandparent Process (R CMD check): Captures the double-mojibake, assumes it is KOI8-R, and converts it a third time.

If we reverse the triple-encoding of the string in your log (п©ц╥я▐Б■─...), it translates perfectly to the standard R error:

"Error in parse(con, keep.source = FALSE, srcfile = NULL) : invalid input found on input connection..."

aitap · 2026-08-06T10:15:42Z

            }
          }
-        shell: Rscript {0}
+        shell: bash -c 'set -o pipefail; Rscript {0} 2>&1 | iconv -c -t UTF-8'


{0} probably included quotes or something. You can safely pass arbitrary arguments to shell commands like this:

Suggested change

shell: bash -c 'set -o pipefail; Rscript {0} 2>&1 | iconv -c -t UTF-8'

shell: bash -c 'set -o pipefail; Rscript "$@" 2>&1 | iconv -c -t UTF-8' -- {0}

This is so silly. What GHA runner actually runs is not the command line specified in the shell argument, oh no. It splits the shell line by whitespace and makes every word into a separate argument. As a result, the command being run is

bash "-c" "'set" "-o" "pipefail;" "Rscript" '"$@"' "2>&1" "|" "iconv" "-c" "-t" "UTF-8'" "--" "/home/runner/work/_temp/21fe02f1-a9e0-4138-ba8c-875e7bb7469a.sh"

which gives the same error:

-o: -c: line 1: unexpected EOF while looking for matching `''

Naturally, this is not documented at all.

That approach is kinda doomed because this is shared with windows runners too, right?

Quite.

I'm testing a solution in 1e27ce0 (https://github.com/Rdatatable/data.table/actions/runs/31110427909), limiting the time spent running with a non-default locale to R CMD check only.

aitap requested a review from MichaelChirico as a code owner July 18, 2026 17:17

aitap changed the title ~~GitHub Actions for non-UTF-8 locales~~ GitHub Action for non-UTF-8 locales Jul 18, 2026

MichaelChirico reviewed Jul 22, 2026

View reviewed changes

Comment thread inst/tests/tests.Rraw Outdated

MichaelChirico reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/R-CMD-check.yaml Outdated

MichaelChirico reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/R-CMD-check.yaml Outdated

MichaelChirico approved these changes Jul 22, 2026

View reviewed changes

MichaelChirico added ci encoding issues related to Encoding labels Jul 27, 2026

aitap linked an issue Jul 30, 2026 that may be closed by this pull request

Tests don't handle non-UTF-8 locales well #7217

Open

aitap and others added 3 commits August 5, 2026 22:24

GHA: add locale-specific checks

d20dd3c

Drop unnecessary environment variables

c2aeade

Put encoding configs in R-CMD-check-occasional, enabling them for now

e5b3b62

MichaelChirico force-pushed the GHA-locales branch from 060f490 to e5b3b62 Compare August 5, 2026 22:30

MichaelChirico added 2 commits August 5, 2026 23:17

leave ISO-8859-1 implicit for fr_CA

0a3723f

Gemini: iconv to prevent gibberish in GHA logs

b465638

MichaelChirico force-pushed the GHA-locales branch from f7e51f8 to 3b48dd9 Compare August 6, 2026 06:18

MichaelChirico added 2 commits August 6, 2026 06:29

Gemini: try iconv directly on the 'Check' step

61293b7

Gemini: back to overwriting R binary, but using R.home() for subproce…

2a36892

…sses too

MichaelChirico force-pushed the GHA-locales branch from 3b48dd9 to 2a36892 Compare August 6, 2026 06:35

ben-schwen reviewed Aug 6, 2026

View reviewed changes

ben-schwen approved these changes Aug 6, 2026

View reviewed changes

MichaelChirico mentioned this pull request Aug 6, 2026

Support more locales in R-CMD-check-occasional #7869

Merged

aitap commented Aug 6, 2026

View reviewed changes

	shell: bash -c 'set -o pipefail; Rscript {0} 2>&1 \| iconv -c -t UTF-8'
	shell: bash -c 'set -o pipefail; Rscript "$@" 2>&1 \| iconv -c -t UTF-8' -- {0}

Uh oh!

Conversation

aitap commented Jul 18, 2026

Uh oh!

codecov Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelChirico left a comment

Choose a reason for hiding this comment

Uh oh!

MichaelChirico commented Jul 27, 2026

Uh oh!

aitap commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelChirico commented Jul 28, 2026

Uh oh!

aitap commented Jul 28, 2026

Uh oh!

MichaelChirico commented Aug 5, 2026

Uh oh!

ben-schwen Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

ben-schwen left a comment

Choose a reason for hiding this comment

Uh oh!

aitap Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aitap Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelChirico Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

The Recursive Mojibake (Why the text is so garbled)

Uh oh!

aitap Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aitap Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

MichaelChirico Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aitap Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Jul 18, 2026 •

edited

Loading

aitap commented Jul 28, 2026 •

edited

Loading

aitap Aug 6, 2026 •

edited

Loading

MichaelChirico Aug 6, 2026 •

edited

Loading